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transition programs, design issues, context issues, strategies for 
data collection, and analysis and reporting. Examined in the section 
on impa t evaluation are the following issues: outcomes and their 
measurement, comparisons, degree of implementation and student 
participation, data collection, and analysis and reporting. In the 
final section, the tasks entailed in evaluation management are 
described, and suggestions for evaluation managers are presented. 
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Foreword 



As we approach the twenty-first century, a large segment of our nation's young people 
are having a harder and harder time moving from school to work with any reasonable 
prospect for long-term productive employment. The lack of a comprehensive and 
effective school-to-work transition system not only frustrates many students but also 
has substantial costs to business and to our economy as a whole. A skill-deficient 
work force hampers our nation's economic growth, productivity, and ability to com- 
pete in an international economy New modes of information and technology have 
forced a restructuring of the home, the school, and the workplace. As a result, there is 
a critical need to create systems that effectively serve the interests and potential of 
young people who are not planning to enter college directly after high school. These 
students need to leave school with the diverse skills, knowledge, abilities, and attitudes 
necessary for a rapidly changing world of work; community, social, family, and adult 
responsibilities; and lifelong learning. 

The School-to-Work Opportunities Act of 1994 offers a chance to bring together 
partnerships of employers, educators, and others to build an effective school-to-work 
system that prepares young people for either high-quality jobs or further education 
and training. The new system; must include the following basic program elements: 

» work-based learning that provides a planned program of job training or expe- 
riences, paid work experience, workplace mentoring, and instruction in gener- 
al workplace competencies 

• school-based learning that provides career exploration and counseling, instruc- 
tion in a career major, and a program of study that is based on high academic 
and occupational skill standards 

° connecting activities that bring schools, students, and employers together to 
connect the worlds of school and work by matching students with work-based 
learning opportunities and by training teachers, mentors, and counselors 

The challenge is to build and implement a new system that moves beyond business as 
usual for students who are not on the college path. Their transition process from 
school to work must become the coordinated responsibility of school, family, business, 
community, and government. No single institution can or should take sole responsi- 
bility for or be expected to provide all of the approaches to educating, training, guid- 
ing, preparing, and supporting our young people. 
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The Academy for Educational Development's National Institute for Work and 
Learning has undertaken a Study of School-to-Work Transition Education Reform 
supported by the U.S. Department of Education, Office of Educational Research and 
Improvement. The study focuses on the planning and design, implementation, and 
impact of school-to-work transition reforminitiatives. By documenting the design and 
integrity of exemplary programs and by assessing program experiences and impacts, 
the study offers critical lessons for those interested in adapting or adopting programs 
that effectively link schools v/ith the business community to improve the transition 
from school to work. As part of the study, a series of papers have been commissioned 
to identify critical issues facing practitioners and policy makers as they begin to design 
and implement new school-to-work transition systems. 

The overall study has been guided by a National Advisory Panel, which has provided 
direction and advice on the issues to be explored and topics to be considered. The 
National Advisory Panel comprises the following individuals: 



Paul Barton 
Director 

Policy Information Center 
Educational Testing Service 

Cynthia Brown 
Director 

Resource Center on Educational Equity 
Council of Chief State School Officers 

Jacqueline P. Danzberger 
Director of Governance Programs 
Institute for Educational Leadership 

Sandra Jibrell 

Senior Planning Associate 

Annie E. Casey Foundation 

Anita Lancaster 

Assistant Director 

Defense Manpower Data Center 



Hilary Pennington 
President 

Jobs for the Future 

Franklin Smith 
Superintendent 

District of Columbia Public Schools 

Nevzer Stacey 

Senior Program Officer 

Office of Educational Research 

and Improvement 

U.S. Department of Education 

David Stern 
Professor 

School of Education 
University of California 

Rafael Valdivieso 
Vice President 

Academy for Educational Development 
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This paper accomplishes the following: 

• focuses on recommended standard procedures to evaluate school-to-work tran- 
sition programs 

• examines both process and impact evaluation 

• places evaluation as a critical element in overall program management strategy 

• offers evaluation objectives, program features to be measured, and recom- 
mended procedures for conducting and incorporating evaluation into the pro- 
gram development and operation cycle 

• presents the following set of critical dimensions of process evaluation: 

implementation of the program model 
challenges to implementation 
design issues 

implementation processes 
context issues 

data collection, analysis, and reporting 
0 discusses the following dimensions of impact evaluation: 
outcomes and their measurement 
comparisons 

degree of implementation and student participation 
data collection, analysis, and reporting 

School-to-work program models are used as examples throughout these discussions. 

The issues and information provided in this paper are important for those at the state 
and local levels with responsibility for school-to-work opportunities. As the paper 
points out, evaluation, as an integral part of the project management cycle, is critical 
to successful implementation of a schooi-to-work system. 

Ivan Charner 

Vice President and Director 

National Institute for Work and Learning 

Academy for Educational Development 



Introduction 



Our existing school-to-work transition system has been severely criticized in recent 
years because of how poorly it serves our economic growth and development (by 
preparing insufficient numbers of adequately trained youth) and because of how poor- 
ly it serves a large portion of our adolescent and young adult population, particularly 
those with more limited economic and social resources. These problems are character- 
ized by the increasing length of time several groups of youth require in making the 
transition to full-time primary employment. Many youth, particularly school 
dropouts and minorities, may flounder in the labor market throughout their twenties 
before they find stable employment. At the same time, business leaders and policy 
makers have become increasingly concerned about the availability of qualified young 
workers, particularly as demographic projections show that an increasing percentage of 
young people will come from poor and minority backgrounds (who historically have 
been less well educated and less prepared for employment). 

The increased emphasis on new technology and the need for collaborative work styles 
increase the demand for even better-prepared young people for entry-level positions 
than projected by current demographic trends. The projected link between worker 
skill availability and future economic competitiveness and strength has made the 
improvement of school-to-work transition programs and services national and local 
priorities. 

Developing a more effective school-to-work transition system requires changes in pro- 
grammatic practice and the structure and delivery of curriculum and related services 
in our public schools. Kazi- suggests two ways to pursue these changes: 

• encouraging experimentation and learning from new models 

• setting in place the building blocks for a system that focuses on performance and out- 
comes and can therefore accommodate diverse program models (1993:16) 

New and existing models should be systematically evaluated in the following areas: 
their implementation, their outcomes, their relation to the intended objectives for 
improved economic preparedness, and their transition capability to employment for 
our young people. 

High-quality comprehensive school-to-work transition program models include career 
academies (Stern, Raby, and Dayton 1992), Tech Prep programs (Hull and Pamell 
1991), youth apprenticeship programs (Jobs for the Future 1991), cooperative education, 
and career magnet schools, among others (Crain, Heebnei, and Si 1992). While they 




differ somewhat in their structure and targeted student population, these programs 
share the common, goal of preparing high school students for meaningful employment 
and the successful progression from high school graduation to work (U.S. Govern- 
ment Accounting Office 1993). These programs share many core features: 

0 applied academic courses 

° a focus on a selected business or industry (such as financial services, printing, 
or health care) 

• career exposure and preparation 
° preemployment preparation 

• sequenced vocationally related skills training 

0 linkage between academic and work-based or vocational instruction 
0 work-based learning 

• extensive business and industry involvement in program development and 
operations 

• supportive services, including career counseling and job referrals 

These are complex program models with diverse participation from various con- 
stituencies. First, they entail multiple partners, usually the school and one or more 
businesses. Depending upon the scope and purpose of a program, they can also 
include higher education institutions, labor, intermediary business agencies, and com- 
munity-based organizations. Each agency can perform a range of functions within the 
program, such as development, oversight, provision of work experience for students, 
curriculum development, ind staff training. Thus, a program exists through a collab- 
oration, which in itself requires nurturing, management, and evaluation. Such pro- 
grams are usually funded through a mixture of direct and indirect funds and in-kind 
contributions, further complicating their management and operations. Finally, these 
programs operate within the existing policies and procedures that define what is 
appropriate for a high school education (including the minimum standards that must 
be met) and the limits for youth employment (in terms of maximum weekly hours 
and type of work that can be performed). 

Yet, only limited evaluations have been completed to demonstrate how best to imple- 
ment these programs and show their significant contribution to facilitate students' 
transition to successful career-track jobs and continued training, particularly in con- 
trast to more traditional classroom-based training. While several large-scale impact 
evaluations are currently under way for some program models, it will be several years 
before their results are available (Pauley et al. 1994). 

This paper outlines recommended standard procedures to evaluate school-to-work 
transition programs specifically, either conducted b/ program staff (as an internal 
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evaluation) or by an independent agency (as an external evaluation). Ongoing assess- 
ment, feedback, and evaluation are as important as program design and management 
procedures. When incorporated into an overall management strategy, evaluation serves 
an important function in providing feedback on a program's implementation and its 
impact on targeted students, staff, school, and community, including participating 
employers. To serve this function effectively, however, evaluation feedback mecha- 
nisms must be incorporated into program design and operations; the results must be 
reviewed at relevant intervals; and the findings must become the basis for program 
modifications, improvement, expansions, public relations, and publicity, as relevant. 

New federal legislation, the School-to-Work Opportunities Act of 1994, was designed 
to hasten state and local area development of school-to-work transition programs and 
systems. State and local areas will need to draw upon existing evidence of program 
effectiveness as they develop their own plans and will need to incorporate evaluation into 
their development efforts for assessment of their implementation and outcome results. 

Evaluation encompasses several different activities and is conducted for a variety of 
purposes. The primary purpose of evaluation is to provide information on how well a 
program and its components are operating and whether these are yielding the intend- 
ed outcomes. These two objectives represent process (or formative; evaluation and 
impact (or summative and outcome) evaluation, respectively. Process evaluation 
addresses how well a program model, as designed, has been implemented, what types 
of problems are related to implementation, and the factors that support implementa- 
tion.. Impact evaluation addresses how effectively a program achieves its goals and 
objectives for targeted students and the broader school and business communities to 
be served. 

While process and impact evaluations are designed for separate purposes, they are nec- 
essarily interdependent. A program's impact cannot be effectively determined without 
first ascertaining whether and how well a program and its various components have 
been implemented and the degree to which environmental and contextual factors 
interfere with its operations and potential effectiveness. An impact evaluation needs 
to isolate the effects of a program on student outcomes, separate from other possible 
influences and reasons for the outcome or result. Ineffective implementation of a pro- 
gram would need to be corrected before a useful impact evaluation could be undertak- 
en. Conversely, feedback on the outcomes of a program can be used, in turn, to 
investigate implementation and operational issues, particularly if the intended results 
are not attained. To some degree, process and impact evaluation processes can be 
charted as stages in a progression for program development, management, and subse- 
quent replication or adaptation, as the diagram below shows. 
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Program Development and Evaluation Cycle 

While process and impact evaluations are shown as a cycle in the following diagram, 
these stages may, in fact, feed back on themselves, depending on the operational prob- 
lems encountered and the need for program modifications before continuing. All of 
these processes are closely linked to program management, which can make use of 
evaluation findings for the next steps in program modification, refinement, or 
redesign, as necessary to achieve the intended outcomes. 

Below is a discussion of process and impact evaluation. Included are evaluation objec 
tives and program features to be measured and recommended procedures for conduct 
ing an evaluation and incorporating it into program management. The first step in 
conducting either type of evaluation, however, requires clarification of a program's 
goals and objectives. These are the standards by which the degree of program imple- 
mentation and impact are to be measured. 

Program Development and Evaluation Cycle 




Program Design 
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Program Goals and Objectives 



Determining what to evaluate in a program requires a substantive knowledge of a pro- 
gram's goals and objectives. Clearly stating goals and objectives for a program is an 
art, however. According to Rossi and Freeman, goals are generally abstract, idealized 
statements of desired outcomes (1993). For example, a school-to-work transition pro- 
gram's goal would be to improve the quality of the entry-level work force for growth 
industries. 

Objectives, in turn, are the operationalized form of these goals. As Rossi and Freeman 
explain further, sound objectives spell out in detail the condition to be dealt with and 
one or more measurable criteria for a program's success. Good program objectives are 
specific, relevant, measurable, and attainable. Too frequently, however, program goals 
and objectives are described only in broad terms, such as their relation to larger social 
problems (e.g., the program will prepare youth for the technological challenges of the 
twenty-first century), rather than in terms of what the program will specifically 
accomplish (e.g., students will be trained for entry-level positions in computer repair, 
or students will develop problem-solving and group-process skills, as is increasingly 
required for entry-level employees). 

Comprehensive school-to-v/ork transition programs will need to cover at least some of 
the following types of objectives: 

• to teach students about the content and field of a business or industry 

9 to improve students' academic skills, particularly as they relate to the needs of 
the targeted business or industry 

• to develop students' employability skills 

° to improve students' school attendance, academic achievement, and timely 

completion of high school 
0 to train students in specific job-related or vocational skills 

• to train students to be comparable to or better than entry-level employees for 
a targeted business or industry 

0 to encourage graduates to pursue further education and employment in 
promising careers in the targeted business or industry 

• to facilitate a successful and timely transition from high school to employment 
or further postsecondary training 



A program's goals and objectives become the standards for assessing the effectiveness 
of a program's implementation and the adequacy of the outcomes yielded by the pro- 
gram. In terms of process evaluation, the goals and objectives will be used to deter- 
mine whether a program's design is relevant and the degree to which program imple- 
mentation will address these objectives. For impact evaluation, the outcomes should 
be framed and measured according to the intended objectives, and the results should 
be presented according to how well the program is attaining its objectives. 



Process Evaluation 



The first fecus for evaluation of a program model is its implementation. There are 
two dimensions to process evaluation. The first dimension examines the program 
model and what happens to it during the implementation process and determines the 
adequacy of the program model, its operation, and the extent to which the program 
model has been altered through implementation and adaptation. Shipman (1989) 
recommends a generic framework used by the U.S. General Accounting Office 
(GAO) for process evaluation of a program model. The essential elements of a pro- 
gram model that GAO recommends be documented include the problem being 
addressed by the program; the program's purpose, goal, and objectives; program oper- 
ations; administrative structure; funding; and participation levels. 

The second dimension examine* the implementation process itself, focusing on how 
the program's design relates to the organization's need, capacity, and understanding of 
the program's purpose; the organizational factors that support the change process 
(including readiness, resources, and the role of key actors in facilitating the change 
process); the organizational response to program operations through necessary restruc- 
turing and ongoing monitoring and problem solving; and finally, the organizational 
context for the program's implementation. 

Implementation of the Program Model 

Beginning with the first dimension, delineation of a program model is useful in iden- 
tifying its essential features and how these (particularly program operations, adminis- 
trative structure, and funding) operationalize its goals to address a specific social 
problem. In addition, an assessment of service delivery levels and use of each program 
feature may pinpoint the stronger and weaker elements of a program model. 



Key Elements of the Program Model and 
Implementation Process to be Assessed in a Process Evaluation 



Essential Elements 

Program Design 



Program Mode 

Essential program features 

Adequacy of the program 
model 



implementation Process 



Needs being addressed 

Shared vision about the pro- 
gram's intent and purpose 

Scope of the change required 



Organizational Characteristics 



Program Operations 



Operational efficiency 
and effectiveness 

Program fidelity 

Program coverage in reaching 
intended audience 



Organizational readiness 

Sufficiency of resources and 
time for the change process 

Role and supportiveness of key 
organizational officials and staff 

Monitoring and problem coping 

Organizational restructuring 

Ongoing staff development and 
resource assistance 



Context 



Relation to other programs 



Organization's track record for 
change 

External resources 
Government initiatives 



Extensive research has demonstrated that a program design can be dramatically altered 
in the transition from its proposed design to what is actually implemented (Fullan 
1991). This problem is a lack of program fidelity, which Shipman defines as whether 
a program has been implemented and is operated as intended and in "conformance 
with accepted professional standards" (1989, p. 24). The degree of program fidelity 
should be measured as a dimension of program implementation. 

Several problems and circumstances contribute to changes in a design during imple- 
mentation and thus the program model's fidelity: 

• difficulties with the design (such as ambiguous or unworkable components) 
° difficulties with the setting (including the lack of resources or 
insufficient structure) 
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» staffing problems and limitations (lack of staff, insufficient training, and other 

staff preferences for the program) 
° timing (constraints on when and within what time frames a program can be 

implemented) 

9 difficulty in serving a target population (recruitment challenges, competing 
opportunities) 

" the method for implementing the program (in increments or all at once, num- 
ber of sites in which it is implemented simultaneously) 
• support and management of the implementation 

Another operational dimension, the administrative efficiency of a program, should 
also be assessed. Shipman defined this as "the extent to which program resources are 
efficiently managed or expended" (1989, p. 24), including management and quality 
control procedures. 

Contribution of a school-to-work transition program will depend upon its administra- 
tive efficiency. Such programs are often developed through a mixture of funding and 
resources and through multiple collaborations between schools and businesses (and 
sometimes labor, higher education institutions, and community-based agencies). All 
vested parties will need assurance that their contributed time and resources are well 
used, creating pressure for efficient management and oversight practices. In addition, 
since such programs are often funded through multiple sources and in-kind contribu- 
tions, these resources must be carefully managed to maximize their use and impact 
and to reinforce the effectiveness of these various investments. 

A program's interrelationships with other programs should also be measured. 
Shipman defines this as "the nature and extent of relationships between this program 
and other programs and [the] constraints or advantages . . . [created] for the program" 
(1989, p. 24). School-to-work transition programs exist within the regular school 
environment and overlap with the existing courses and activities. Similarly, to the 
degree that these programs involve business and industry, they also overlap with ongo- 
ing business operations and other forms of business preparation in education. All 
these programs, operations, and services provide a context for the school-to-work 
transition program, determining the competing demands on school and business 
preparation, and, specifically, competing employment preparation programs, which 
may overlap or complement the school-to-work transition program. 



General Challenges to Implementation 

Implementation of a program entails some form of organizational change, which is 
inherently a social process. Much research has been directed to understanding how 
and why organizations, particularly schools, respond to program change and imple- 
mentation, providing greater insight into potential problems and common challenges 
for successful implementation. The primary factors to be considered in the imple- 
mentation process are the need being addressed, the scope of the change required, the 
readiness of the organization for change, the role and support of key persons within 
and among participating organizations (such as the staff", program director, school 
principal, district office officials, and collaborating business officials) in the implemen 
tation process, sufficiency of resources, the total time available for the change process, 
and the environment within which the change occurs. 

Fullan also points to critical internal and external factors and themes for implementa- 
tion. These terms are useful in identifying what may promote or inhibit implementa- 
tion and why. Internal factors refer to the factors within the organization that moti- 
vate the change; the school and district's track record with changes; and the key 
actors' support, cooperation, and knowledge about the program being implemented. 
A school's or district's experience with school-to-work transition programs and ser- 
vices and the general degree to which the institution is experimental influence pro- 
gram implementation. The external factors refer primarily to the government man- 
dates and external resources that may define how a program can be implemented or 
operated. In the case of a school-to-work transition program, these mandates can 
include child labor laws and high school graduation requirements. 

The key themes as outlined for implementation by Fullan point to important process 
es in the development and support of a program's successful implementation: 

9 vision building (which is the degree to which the program philosophy and 
approach are well understood and shared by those in the program and in the 
school and district) 

• evolutionary planning (using both top-down and bottom-up planning process- 
es throughout a program's development and implementation) 

• empowering program staff (the more latitude staff have to make decisions 
about a program and its implementation during the development and start-up 
process, the more successful its implementation is likely to be) 

0 developing staff and resource assistance (having sufficient resources and initial 
and ongoing staff development and technical assistance available improves 
chances of the program's success) 



0 monitoring and problem coping (systematic assessment and review of practices 
and mistakes helps to redesign a program to be more effective) 

0 restructuring (acknowledging that the new program may have an impact on 
the school's policies, roles, governance, and even finances and permitting time 
and planning to consider these types of impacts and how best to deal with 
them) 

As part of a process evaluation, it is useful to examine how problems and roadblocks 
are handled by assessing the following: 

» avoidance or successful resolution of implementation process- related problems 
that could undercut the program's effectiveness 

• the capacity to overcome the limitations of the setting and other constraining 
circumstances thar surround the program 

implementation Challenges for School-to-Work Transition Programs 

In addition to the general implementation problems and challenges faced by any pro- 
gram new to a school or district, some implementation problems are typical to com- 
prehensive school-to-work transition programs. These implementation challenges 
commonly center on the following: 

• selection, development, and implementation of new curriculum, teaching 
methods, assessment strategies, and certification 

• need for structural changes within the school — such as adding new courses, 
integrating academic and vocational instruction, adding work-based learning 
experiences, altering scheduling, and allowing off-site experiences — all of 
which challenge the school's and district's flexibility, support, and use of staff 
development 

0 business participation and involvement that entails the challenges of collabora- 
tion and varied roles in program governance, instruction, work site experi- 
ences, and resource sharing 

0 ways in which the program design may challenge state and local education 
regulations and, for workplace training, child labor laws 

e issues of equity and access, addressing particularly the experiences and out- 
comes of selected subgroups such as minorities, young women, limited 
English-speaking students, and students with disabilities (Kazis and Niles 1 992) 
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These implementation challenges and how they can be assessed are described below. 
Specifically, the process evaluation issues center on design, implementation, context, 
data collection, and analysis. The career academy model, as developed and supported 
by the National Academy Foundation, will be used as an example in clarifying these 
issues and challenges. 

Design Issues 

Rarely are programs implemented as a perfect replication of the original or intended 
model's design, which makes assessment of program fidelity an important analytic 
process. Some adaptations to local conditions are beneficial. Some adaptations may 
affect less important components of a model, thereby not undercutting its potential 
overall effectiveness; other forms of adaptation may severely alter the program so that 
it bears little resemblance to the original design and is unlikely to address its intended 
goals and objectives. 

For example, the career academy model is typically a two-year program of coursework 
with a summer internship in between the two academic years. Career academies focus 
on a selected business or industry sector. Some districts have adapted the career acad- 
emy model into a four-year program, providing high-risk students with preparatory 
coursework that will help them succeed in the core program courses and internship. 
Such a modification is consistent with the program model and extends its benefits to 
serving high-risk students. Some districts, however, have difficulty arranging for 
internships. Deleting the internship component would be an adaptation that would 
severely alter the program ?.nd make it unlikely to meet its intended objectives of 
preparing youth well for the workplace. 

The characteristics of a program design and the way these are explained can have bear- 
ing on the fidelity of how a program model is implemented. Some features of a pro- 
gram's design can inhibit successful implementation of the model as intended because 
they are challenging or require special circumstances to be implemented. For exam- 
ple, some features may be too complicated to permit easy implementation, leading to 
adaptation of a feature or deletion of a component. 

Sometimes program designs have broad goals and objectives that are ambiguous and 
can confuse program officials about the intended thrust of program implementation. 

In addition, some features of a program model may not be well defined by the 
model's designers, leaving too much to program operators to decide and possibly mis- 
interpret. For example, one program objective may be to integrate academic and 
work-based learning. This process entails many strategies, from exposure to work- 
related content to actual work-related skill development in academic coursework and 
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the reinforcement of specific academic skills at the workplace. Without specific clarity 
and guidelines, program operators could develop a program that gives only cursory 
attention to this integration of skills development. 

Finally, some features may be controversial for the local school or setting, forcing oro- 
gram operators to drop or adapt them. For example, school staff may be reluctant to 
alter the school schedule to permit block programming for work-based training or to 
group students for a series of common courses. 

Some program components are dependent upon appropriate local conditions. In par- 
ticular, local districts should focus their school-to-work transition programs on growth 
industries, where training opportunities could best be developed and subsequent 
employment opportunities would most likely be for program graduates. For example, 
if a business or industry in a particular community is experiencing a downturn or 
excessive layoffs, such as occurred in the computing industry during the late 1980s 
and early 1990s, a career academy in computing should not be initiated in that com- 
munity. S'milarly, local conditions may make it impossible to implement a program 
model fully or with adaptations that are consistent with the goals and objectives. 
Rural school districts, for example, may not be able to arrange adequate work-based 
learning opportunities in targeted businesses and industries for a career academy pro- 
gram, forcing district officials to forgo this aspect of the program or to substitute 
other experiences, which may not sufficiently simulate actual work or the industry. 

Evaluation of adherence to a program's design through implementation entails deter- 
mining the core components of the model, the extent to which each component is 
actually implemented, the degree to which a program's students participate in ali 
aspects of the program, and the consistency of adaptations with the program's pur- 
pose, goals, and objectives. 

Implementation Processes 

Even with a well-delineated model, programs can encounter problems because of how 
they are implemented. According to Fullan, there are three distinct phases to the 
change process — initiation (which includes the steps leading up to the decision to 
adopt a program or model), implementation (which entails the first efforts to put the 
program ideas into practice), and continuation (during which a program becomes 
institutionalized as part of the system or discarded). The change process, while 
including these three phases, is not necessarily linear, particularly as feedback or initial 
implementation efforts may lead to rethinking and modifying a program's design for a 
new round of implementation activity. The amount and source of initial support for 
the change process are critical during these trial periods of planning and initial use. 



Documentation of the implementation process should focus on all three phases, look- 
ing at the breadth, role, and support of those who participate; the training and assis- 
tance provided to staff before and during initial program implementation; and the 
time and resources allotted for each of these phases. There are several operational 
considerations in the implementation process, some that relate to any program imple- 
mentation and some that exist because of the nature of school-to-work transition pro- 
grams. Programs can be implemented in ways that maximize or undercut the poten- 
tial for success. 

Common implementation problems that undercut the program's potential include 
untimely delays in hiring appropriate staff and obtaining curricular materials and 
other resources; starting the program out of sequence with the school calendar so stu- 
dents cannot be recruited adequately; starting the program with only some compo- 
nents and staff in place; insufficient or delayed funding; difficulty in recruiting target- 
ed participants; lack of cooperation from collaborating agencies or other school staff; 
and difficulty in engaging business and industry participation. These problems can 
lead to dramatic adaptations in the program's model and delivery process. It may 
later become challenging for program officials to convert the adapted program over to 
the intended design once operations are under way. 

Context issues 

The characteristics of the school and community within which a program model is 
implemented can affect how the design is implemented and the effectiveness of the 
program in serving youth. The adequacy of resources (within the school district and 
among collaborating businesses) made available for a program and its implementation, 
the setting, institutional and financial support, the existence of competing or comple- 
mentary school programs, and the existence of broader social and political problems 
can have bearing on how well a program is implemented and how it may be adapted. 

For example, program staff and the school and district administration can be severely 
challenged if they must implement several new programs, policies, and procedures 
simultaneously. As well, a school district's fiscal crisis can restrict funds and staff time 
for appropriate planning and program support before and after a program is imple- 
mented, leaving insufficient resources to adequately implement the program. A city's 
economic downturn can upset the viability of targeting selected businesses or indus- 
tries that are adversely affected, undercutting the effectiveness of preparing students 
for that business sector. 
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In examining the context-related issues to implementation, it is important to deter- 
mine the degree to which these factors exist and may be altering the core components 
of the design or undercutting effective implementation. If this is occurring, then 
whether these adaptations are undercutting the program's effectiveness in achieving 
intended student outcomes or whether appropriate solutions have been devised to 
meet these challenges needs to be determined. . 

Strategies for Data Collection 

Evaluation evidence of program implementation can be drawn from program records, 
archival materials, and interviews of key staff and participants. Such data collection 
can occur at key intervals throughout the initiation and implementation process. In 
anticipating a process evaluation, program officials and staff should design record- 
keeping procedures on the program's development and implementation as part of the 
program's management structure. All written information describing the purpose, 
structure, and operations of the program will become critical documentation. In 
addition, a management information system should be designed to track participants' 
experiences in program use through the application, participation, and completion 
phases. This information system can be used to document the number and types of 
participants, retention and completion rates, and service use changes over time. 
Finally, periodic interviews and focus group discussions with key staff and samples of 
participants can provide insights on early start-up problems and possible solutions and 
on subsequent implementation and program operations problems and solutions. The 
following records and archival materials should be maintained: proposal, mission 
statement, program objectives, program description, curriculum, staff development 
materials, staffing structure, minutes of and attendance for key meetings, participants' 
applications and enrollment forms, participant tracking and follow-up forms, and 
public relations materials and news coverage. 

Using a career academy as an example, a prototypical process evaluation design would 
be as follows. Fir:,., the program's goals and objectives would be clarified and appro- 
priate measures of student participation and success would be determined. In this 
case, the program includes eight core courses, a summer internship, and a college-level 
course. A management information system would be used to collect information on 
student participation and success in these courses and internship experience. Its core 
components would include documentation on student eligibility, coursework perfor- 
mance and completion, attendance, and feedback on internship performance. A rat - 
ing form for internship supervisors would be used with a mechanism for timely and 
complete feedback on appropriate internship performance outcomes — employability 
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skills, entry-level knowledge and skills related to the industry, quality of appropriate 
academic skills, and problems with the intern and internship process. 

Through periodic interviews during the initiation and implementation phases, pro- 
gram staff and key school and district administrators would be asked about their 
understanding of the program's goals, objectives, and mission; capacity to implement 
all core components; scheduling of courses and the use of core courses to meet high 
school graduation requirements; provision of space and resources to adequately oper- 
ate the program; and staffing. In terms of the operations of the program, the inter- 
views and review of program records would provide information on the recruiting 
process (and the degree to which the program is being adequately promoted and is 
recruiting targeted students), student orientation, staff training, and internship devel- 
opment. Interviews and observations can be used to determine whether the curricu- 
lum is being used as intended and whether internship experiences are providing stu- 
dents with adequate exposure to the industry and the world of work. 

Analysis and Reporting 

Program officials and staff should establish periodic benchmarks to collect implemen- 
tation evaluation information and analyze the results to inform program operations 
and management practices within a specified time frame. The first step of analysis 
would be to draw up a case summary of the initial development and implementation 
processes. This step helps to integrate all phases and processes into a coherent picture. 
The summary is helpful in drawing out the ways in which the program design and 
structure have been adapted for implementation and pinpointing critical events that 
may have facilitated or hindered optimal implementation. 

The second step is to analyze the implemented program model and processes accord- 
ing to key issues: 

° adherence to the proposed model and ways in which it has been adapted 
0 adherence to recommended or ideal program features and ways in which it 
differs 

0 types of implementation problems, their causes and possible solutions 

By establishing key benchmarks for the development and implementation processes, 
program management and staff can review problems systematically and develop early 
intervention strategies. Reflecting on the program as a whole and the relation 
between objectives and implemented program features is critical to thorough and 
effective problem solving. This allows for more inclusive solutions to be developed, 
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rather than just making incremental decisions as problems arise, and permits early inter- 
vention before these problems have become too entrenched or complicated to resolve. 

Finally, in analyzing the problems and experiences of programs in the implementation 
process, it is valuable to compare the start-up and implementation experiences of the 
program with other similar programs and recommended or ideal program features. 
Such a comparison offers several benefits. First, it may reveal common problems in 
implementing the model, for which other programs may have found solutions. 
Second, such a comparison would help to determine whether implementation prob- 
lems are a function of the model itself or the institutions and program setting 
involved. 

In conclusion, process evaluation is a useful management mechanism to provide feed- 
back on how well a program model is understood and the fidelity with which the pro- 
gram is implemented, to identify operational problems, and to determine whether 
unintended challenges exist that need attention and resolution. A process evaluation 
can become an ongoing management practice and need not be limited to a one-time 
event. Repeated use of process evaluation can help to determine whether a program is 
successfully implemented as intended and can identify when a program is incremen- 
tally drifting from its intended mission and design (as can occur during long-term 
operations). 



Impact Evaluation 



The ultimate objective for evaluation is to demonstrate a program's impact. It is criti- 
cal to determine whether and to what degree a program is meeting its intended objec- 
tives and yielding the appropriate outcomes. In addition, documentation of a pro- 
gram's positive outcomes has several operational and management benefits for pro- 
gram staff and officials. These results can be used to further programs, aid in 
fundraising, advocate for program continuity, encourage participant recruitment, and 
generally provide positive public relations. Therefore, a program's continuation and 
operational success hinge on the successful demonstration of its effectiveness in meet- 
ing or exceeding its objectives. Program officials will want to demonstrate the short- 
and long-term benefits of a program. The quality of these outcome findings depends 
substantially on the appropriateness and effectiveness of the impact evaluation design 
and its administration. 



25 



Determining a program's impact requires clearly stated and measurable objectives or 
indicators of effectiveness and appropriate sources of comparison for these indicators 
to demonstrate the significance of the program's impact on its participants. Ideally, 
an impact evaluation design would incorporate multiple measures of effectiveness for 
each objective or claim, thereby strengthening the validity of the findings. 

The following sections describe outcome measures, comparisons, data collection pro- 
cedures and analysis, and reporting procedures for an impact evaluation of a typical 
school-to-work transition program. 

Outcomes and Their Measurement 

A school-to-work transition program can yield several types of outcomes, the most 
primary being to improve the employability of its youth participants, particularly in 
obtaining and retaining quality jobs with career-track potential in growth industries. 
School-to-work transition programs have basically three types of core outcomes for 
their youth participants: 

• improving participants' knowledge and skills for entry-level jobs in a specific 
business or industry 

• improving participants' ability to pursue postsecondary education in the same 
or a related career field 

° improving participants' ability to obtain timely and promising career-track 
jobs in a specific business or industry 

After appropriate outcomes for evaluation of a school-to-work transition program 
have been determined, relevant and sensitive measures will need to be identified. 
Selected measures should closely reflect the program's mission, design, and service 
delivery. As outlined earlier, three types of outcomes are to be measured: knowledge 
gained, improved ability to pursue postsecondary training, and improved ability to 
pursue career-track potential jobs. Measures of these outcomes must be tailored to the 
specific curriculum and activities of the program and the type of business or industry 
targeted. For example, if the program integrates academic skill development and specific 
business area (such as economics and financial services), an appropriate outcome 
would be knowledge gained about economics and its relation to financial services and 
could be measured through a specifically designed criterion-referenced test. 

Measures of students' improved ability to pursue postsecondary training could include 
college acceptance and enrollment rates (particularly by more challenging postsec- 
ondary programs), as determined through a follow-up survey of graduates; partici- 
pants' pre- and postprogram comparison of self-reported confidence about pursuing 
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and succeeding in postsecondary education; and comparison of teachers' pre- and 
postprogram ratings of the students' capability of being accepted in and successfully 
completing a postsecondary education program. 

Finaily, measures of the program's impact on students' subsequent employment in the 
targeted business or industry could be determined by a short-term follow-up survey 
(such as three to six months after program completion) that assesses their employment 
status (by area of employment) and type of position held, responsibilities of the posi- 
tion, benefits and other remuneration, length of time in seeking employment, and 
assistance used in job hunting. Other measures would be the participants' compara- 
tive sense of confidence (pre- and postprogram) in obtaining employment in the tar- 
geted industry. In addition, if participants have a work experience (like a summer 
internship or other form of work site training), supervisors' ratings of the participants' 
employability skills, applied academic skills, and job-related skills could be assessed in 
general and in comparison to other entry-level employees. 

School-to-work transition programs can also yield several academic and attitudinal 
improvements: 

° improving participants' academic achievement (as measured by course grades 

and grade point averages) 
° improving participants' interest and engagement in school (as measured by 

their affinity for school, attendance, and lack of discipline referrals) 
• increasing the likelihood that participants will remain in and complete high 

school in a timely fashion (as measured by their credit accumulation, retention 

and graduation rates) 
° improving participants' confidence and sense of personal efficacy or general 

capability to succeed in their career pursuits and the world of work generally 

These outcomes represent short-term benefits that result from applied academic and 
work-based skills development and from preparation for employment in a selected 
industry or business. These outcomes can be measured separately for participation in 
program-related courses and activities and for overall school performance and attitude. 

Ideally, a school-to-work transition program should reinforce the Secretary's 
Commission on Achieving Necessary Skills' (SCANS) basic employability skills, which 
represent benchmark preparatory skills for success in the workplace. The SCANS 
competencies and three-part foundation for workers include the following: 
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Competencies 

• resources: worker identifies, organizes, plans, and allocates resources (including 
time, money, materials and facilities, and human resources) 

° interpersonal: employee works with others 

» information: worker acquires and uses information 

0 systems: employee understands complex interrelationships 

• technology: employee works with a variety of technology 

Foundations 

• basic skills: reading, writing, performing arithmetic and mathematical opera- 
tions, listening, and speaking 

• thinking skills: thinking creatively, making decisions, solving problems, visual- 
izing, knowing how to learn, and reasoning 

• personal qualities: displaying responsibility, self-esteem, sociability, self-man- 
agement, and integrity and honesty 

At a minimum, participating students should demonstrate improved competence in 
the various SCANS competencies and foundations. In using these competencies to 
evaluate a program's impact, however, a preliminary assessment is necessary to deter- 
mine how much and to what extent the program emphasizes each skill area before 
ascertaining the degree to which students develop these skills. 

The primary long-term outcome of a school-to-work transition program, however, is 
the participants' subsequent success in obtaining promising entry-level jobs in the 
training-related business or industry, success being that participants who successfully 
complete a program are able to find employment in their training-related field in a 
timely fashion and that the job holds promise of advancement and appropriate remu- 
neration and benefits. Related to this outcome is the degree to which participants 
choose instead to pursue further postsecondary education or training in the same 
career field and are able to enroll and complete such a program before pursuing 
employment in the targeted business or industry. 

School-to-work transition programs can yield secondary outcomes for the staff, 
schools, and businesses and industries involved in operating the program. These 
outcomes include improving school and business relations, teaching practices and 
quality curriculum at the school, and work-site training. This paper focuses primarily 
on participant-related outcomes, which are the priority for school-to-work transition 
programs. 
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Comparisons 

The significance of improvement in the various participant outcome measures and the 
contribution of the school-to-work transition program to these gains can only be 
determined through comparisons with other types of groups' performance on these 
same outcomes. Two types of comparative analysis are recommended for the evalua- 
tion of a school-to-work transition program. The first is a within-program compari- 
son, and the other is a comparison with other groups that did not participate in the 
program. 

The first type of comparison is to determine the degree of program impact on the 
selected outcomes among different types of youth served. The comparative analysis 
would determine whether all types of youth served attain similar outcome results or if 
some types of youth perform less well than others. Possible types of comparisons 
would be by gender, race/ethnicity, and degree of being educationally and economical- 
ly at risk. It is important to ascertain the degree to which the program is equally suc- 
cessful with various subgroups, particularly minority youth and young women, who 
often fare less well in the workplace than do other youth. This comparison is particu- 
larly significant if the related business or industry has historically underemployed spe- 
cific subgroups of youth (such as young women and minorities). 

The second rype of outcome comparison to be conducted is between program partici- 
pants and other groups that did not participate in the program. The objective of this 
comparison is to determine the degree to which il e school-to-work transition pro- 
gram yields its intended outcomes (rather than have these outcomes occur because of 
other nonprogram-related factors) and whether the magnitude of these outcomes is 
greater than what other types of programs (or the lack of any program) can yield. 
There are three types of nonprogram youth to which program participants could be 
compared: (1) similar youth in an alternative type of school-to-work transition pro- 
gram, (2) similar youth in a less comprehensive school-to-work transition program, 
and (3) similar youth who are not in a school-to-work transition program. 

The evaluation challenge is to ensure that any outcome differences yielded by these 
comparisons can be attributed primarily to the program's design, rather than other 
differences in the characteristics of comparison groups. The selection of comparison 
groups must be structured in such a way as to reduce the likelihood that the intended 
outcomes are attributable to other student-related characteristics. The most common 
characteristics to consider are personal demographic characteristics of the youth (such 
a<, age, gender, rau7eihnidty, and poverty status), their prior academic success and 
likelihood of being at risk of dropping out (including their attendance, grade point 
average, credit accumulation, and disciplinary referral histories), and their motivation 



& ii 



to be in the program and the targeted business or industry and to pursue employment 
or more training after graduation in the same or a related career field. Dramatic ini- 
tial differences between participants and comparison group youth in these characteris- 
tics could make it difficult to interpret whether subsequent outcome differences are attrib- 
utable just to the program experience or result primarily from these initial differences. 

Ideally, an experimental evaluation design should be used. Such design, however, 
requires that equally eligible students be randomly assigned to a program group or a 
comparison group, making the two groups identical in their initial characteristics, 
motivation, and interest in the program. This would require overrecruiting eligible 
youth for a program and randomly assigning them to a program or comparison group 
(which would be either an alternative, less intensive, school-to-work transition pro- 
gram or no program). Subsequently, relevant measures of the two groups' achieve- 
ment, employment, and other appropriate outcomes would be gauged and compared 
to determine differences that can be attributed specifically to the school-to-work tran- 
sition program's effects. 

However, opportunities for randomly assigning students and making these compar- 
isons rarely exist. Program officials are reluctant to withhold a program from interest- 
ed youth, even if it is oversubscribed. It can also be politically challenging for schools 
and businesses to use a lottery system to assign applicant students to a program or a 
comparison alternative. Finally, it is often difficult to recruit sufficient numbers of 
youth to permit random assignment to two groups. 

As a result, a quasi-experimental design that uses other means of assigning students to 
comparison groups is usually necessary. These various assignment procedures require 
compromising some of the control on initial group differences between participants 
and comparison students. This type of design, therefore, has some evaluative draw- 
backs because it is more difficult to separate out the influence of the program from 
the initial differences between participants and comparison students. 

Multiple alternative comparison groups could be used to sort out these possible differ- 
ences from the program's effects. The primary initial differences to control are demo- 
graphic and relevant school performance indicators for students in the program and 
comparison groups. Three possible groups are as follows: one group could be drawn 
from participants in a comparable vocational education or school-to-work transition 
program that uses an alternative service delivery design (such as providing only class- 
room instruction in a vocational area); a second group could be youth who applied for 
but were not accepted into the school-to-work transition program because of the lack 
of space, not because of their initial eligibility, and who would therefore be similarly 
motivated to be in the program and have the same demographic profile as participants; 



a third group could be youth of similar background (in terms of demographics and 
school performance) selected from the same or a similar school. 

There are advantages to and limitations of each of these types of comparison groups. 
The first group permits a comparative analysis of two alternative school-to-work tran- 
sition programs and services, with one being more intensive than the other. This 
analysis could help to demonstrate the additionil benefit of the more intensive experi- 
ence. Yet, there may be some differences between the two groups of students, in 
terms of their demographic characteristics and their initial interest and motivation, 
which may account for some of the outcome differences. The other two comparison 
groups permit an analysis of the benefits of being in the program in contrast to having 
no school-to-work transition program experience. Members of the first group would 
be most similar to those in the program because they were similarly motivated and 
would, theoretically, match the demographic profile of those who were accepted. 
Thus, initial differences between the two groups would be minimized. Yet, this com- 
parison group is difficult to construct because programs often do not keep records on 
who is not enrolled, and programs often are not oversubscribed by sufficient numbers 
of equally eligible students to allow for the construction of a comparison group. 
Finally, the third type of comparison group would be easiest to construct, but would 
not permit ruling out initial differences in interest and motivation (which the partici- 
pants might have) and would only allow comparison of the program's impact with a 
nonprogram experience. 

Determining the program's impact on various outcome measures can be strengthened 
by using multiple types of comparisons. This includes using two or more locally con- 
structed comparison groups, drawn from the above types. This also includes using 
other types of comparative benchmarks, such as district, state, or national level data or 
hypothetical standards based upon a rater's professional experience. For example, pro- 
gram participants' school retention and graduation rates can be compared with a simi- 
lar group of students in a general education program, but also to the districtwide aver- 
ages. Survey results from a nationally representative sample of students, such as those 
yielded from the National Educational Longitudinal Survey of 1988 and the subse- 
quent follow-up surveys, permit calculation of student subgroup profiles on selected 
school-related attitudes and performance measures. These data could be used as a 
national comparison on the program's impact. Finally, relevant experts, such as pro- 
gram teachers and work-site supervisors, can rate program participants' performance 
on selected outcome measures in relation to other types of students or employees. For 
example, work-site supervisors could rate the participants' performance relative to the 
general performance level of other entry-level employees. 
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Degree of Implementation and Student Participation 

A measure of the degree to which youth participate in the various program elements 
of a school-to-work transition program is necessary for analysis and interpretation of 
findings. Many school-to-work transition programs contain multiple features: acade- 
mic courses, preemployment preparation, work experience or work-based learning, 
exposure to role models and mentors, and supportive services. Participant outcomes 
should be positively related to the length, the intensity of exposure to these compo- 
nents, and the number of components experienced. That is, the longer a youth par- 
ticipates and the more services he or she uses, the more positive the outcomes are like- 
ly to be. For example, career academy students who participate in the full two-year 
program and have an internship should have higher program outcomes than students 
who have only the two-year program, but not the internship, or students who partici- 
pate in only one of the two program years. 

This measure of students' participation in the program would be based upon partici- 
pants' attendance and total use of all program services and components. It would be 
used in the analysis to differentiate participants (by degree of program usage and par- 
ticipation) and compare the outcome results. This analytic step will help to demon- 
strate the degree to which level and intensity of program participation contribute to 
the outcome results yielded. 

Data Collection 

Data for an impact evaluation can be obtained through qualitative and quantitative 
data collection techniques and procedures. The data collection strategies to be used 
depend upon the outcome measures selected, the availability of information for these 
measures, any existing constraints on data collection (which may limit the degree to 
which various data collection techniques can be used), timing, and available resources. 
A wide variety of data collection procedures exist and can be used singularly or jointly, 
depending upon the evaluation design. These procedures can include analysis of pro- 
gram documentation or participants' attendance, completion, and performance; 
observations of activities; interviews; and follow-up studies of subsequent employment 
and educational experiences. Since it is advisable to use multiple measures of each 
program outcome, it is likely that a variety of data collection procedures will be neces- 
sary for the impact evaluation. 

Data on participants and comparison group youth must be collected in similar, sys- 
tematic procedures for the same time intervals and according to the same schedule. 
Ideally, the evaluation design should be to collect school performance information on 
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both participant and comparison group students for their status prior to enrolling in 
the program and at regular intervals during and at the end of the program. 

Some data collection can be excerpted from existing program record keeping. 
Conversely, some data can be collected by incorporating new categories into existing 
program record keeping and documenting procedure. Finally, some data collection 
will have to be undertaken specifically for the evaluation. In these cases, it is advisable 
that the timing of these collection activities be tailored to program operations and 
scheduling to facilitate ease in data collection and to increase the probability of 
obtaining valid and reliable information. 

Some outcome measures can be documented as part of an existing management infor- 
mation system on service delivery and student participation. Through advance plan- 
ning, data collection for an impact evaluation can be incorporated into the ongoing 
management practices of a program to permit easy and timely access of information 
and reporting. For example, enrollment applications could be used to collect prepro- 
gram measures of students' attitudes about school and work, their future career and 
educational plans, attitudes about themselves and their capacity to find successful 
employment, and their prior relevant experience. Other data, such as grades, atten- 
dance, disciplinary referrals, achievement test performance, and school completion, 
can be obtained through an existing school district management information system. 
A school district's management information system can also be used to collect infor- 
mation on participants' school-related performance prior to enrollment, their academ- 
ic progress during program participation, and their academic achievement at the com- 
pletion of the program, and similar information for comparison students. 

The program's management information system could be used to document student 
participation (such as attendance, progress, and accomplishments) in the various 
program components. If comparison groups are used, a separate management infor- 
mation system may be necessary to track the academic progress and other similar pro- 
gram experiences of these youth if they are receiving other forms of employment 
preparation or vocational training. 

Youths' participation in the program can be documented further through periodic 
observations of them in selected program activities. Through observations, an evalua- 
tor can document attendance patterns, student interest and engagement, student- 
teacher (or work-site supervisor) interaction, use of instructional materials, and the 
overall climate and tone of the program. At a minimum, a sample of classes and work 
sites should be observed at least once, but preferably twice or more at well-spaced 
intervals during program operations in a program year. 
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The benefits and influences of the students' program experiences can be ascertained 
more fully through individual and group interviews, particularly focus group inter- 
views. According to Kreuger (1988), focus group interviews are useful in gaining 
insight, perceptions, and explanations of experiences from participants. Focus groups 
make use of the group environment to promote self-disclosure among participants. 
Focus groups are typically comprised of seven to ten people of similar backgrounds 
who are led through a small number of open-ended questions and related discussions 
by a group facilitator. The format allows for follow-up probes on answers. These in- 
depth interviews are useful in developing an understanding of a program from the 
participants' perspective and can provide more richness of information than is attain- 
able from surveys. Other qualitative data collection procedures can include construct- 
ing case histories of a small sample of participants (based on a series of in-depth inter- 
views) to learn how the program fits into their life experiences and future plans. 

Surveys can be a useful data collection mechanism to collect standardized information 
on participants' attitudes, program experiences, future plans, and evaluation of their 
program experiences. To measure change in attitudes and expectations as a result of 
the program, pre- and postprogram surveys can be administered to participants and 
comparison group students. Several nationally developed surveys, such as the 
National Education Longitudinal Survey (sponsored by the U.S. Department of 
Education), can be a source of well-developed questionnaire items. 

A follow-up survey of program graduates conducted at six months, twelve months, 
and forty-eight months after program completion and high school graduation can be 
used to measure the short- and long-term benefits of the program in preparing youth 
for employment and further postsecondary education and training. A similarly con- 
ducted survey of a comparison group can help to demonstrate the comparative bene- 
fits and gains of the program for its graduates. 

Analysis and Reporting 

The analysis and reporting of impact evaluation results should be undertaken according 
to the stated program objectives and intended outcomes. This will permit comparison 
of findings on individual outcomes across various sources of data and measures. Once 
these outcome-specific analyses have been conducted, a more general analysis across 
outcomes can be done to determine the program's overall effectiveness and to reflect 
on the relation between the results and the program design, and operational reasons 
for the program's success or lack of success can be taken into account. 



Four levels of analysis should be considered. The first level of analysis is to examine 
the relationship between the outcome results and the educational assumptions of the 
program model itself. This level of analysis is to examine the outcomes as they are rel- 
evant to the design and intentions of the program. If, for example, the program is 
designed to integrate work-based learning with academic instruction (such as having 
students learn about the financial services industry as part of an economics course in a 
career academy program), the degree to which students actually learn about the finan- 
cial services industry as well as economics should be analyzed as part of the evaluation. 
A comparative analysis of student performance in a traditional economics course with 
the performance of participants in a work-based economics course should help to 
demonstrate whether this contextualized instructional approach, as thz program 
design assumes, actually yields better outcomes. 

The second level of analysis is to investigate the relationship between program impact 
and the degree of program implementation (particularly if the program is in multiple 
sites) and the degree to which participants are exposed to and use all aspects of the 
program. There should be a direct and positive relationship between the intensity and 
amount of program offered (and implemented) and used by students and the out- 
comes achieved. 

This is actually two forms of analysis. The first part is to determine whether there is a 
relationship between degree of program implementation and outcomes. That is, if 
one or more program components was not implemented, thr n the analysis should not 
yield any significant results related to this component. For example, if a career acade- 
my program did not provide an internship experience for all participants, less positive 
employment-related outcomes may be expected. If a program model was implement- 
ed in multiple sites, the outcomes of the program should be compared among sites 
according to the degree to which the sites implemented the full program model. Sites 
with more fully (and well) implemented programs, in keeping with the intended 
model, should yield more significant outcomes than programs that are less well or less 
fully implemented. 

In the same vein, individual student outcomes should vary according to the degree to 
which they participated in the full program. Therefore, an analysis of the outcome 
results should compare the outcome levels of students according to their degree of 
attendance and use of all program components. Students who had good attendance, 
participated in all components, and completed the whole program should have better 
performance-related outcomes than students who dropped out of the program early or 
who completed some components and not others. If such differences do not exist 
(and the lack of difference cannot be explained by other influences, such as differences 



35 



in student characteristics according to the degree of program use), the program model 
may not be effective. An underlying assumption of the program is that all compo- 
nents are necessary to yield the intended outcomes. If similar outcomes can be 
reached without full program use, the program design may need to be reexamined. 

The third level of impact analysis would be to demonstrate the comparative benefit of 
the program over other alternatives by contrasting the experiences and outcomes of 
participants and of students in the comparison groups. This analysis would statistically 
compare the results of participants and one or more types of comparison students 
according to the intended program outcomes. The ways in which the two groups dif- 
fer in their training and educational experiences should be reflected in the type and 
degree of outcome differences. For example, if career academy participants are com- 
pared with students who have only taken economics coursework and had no other 
employment exposure, the participants should show somewhat better knowledge of 
economics outcomes and even better employment readiness and ease in finding post- 
high school employment and in pursuing concrete postsecondary educational plans. 
This comparative analysis should show the statistically significant contribution the 
program made to the participants served. Thus, the degree to which the training and 
education experiences of the participant and comparison groups differ should be 
reflected in me outcomes. 

The fourth level of analysis would be within the program to determine whether 
different subgroups of youth have equivalent experiences and outcomes. This type of 
analysis is important to determine whether groups that are traditionally underserved or 
who have limited access to the business or industry (such as young women, minorities, 
and at-risk youth) gain similar or even better outcomes than other program participants. 

In analyzing the program outcomes, it is important to consider the degree to which 
the findings are influenced by other external factors. The primary objective of the 
outcome analysis is to determine the impact of the program, separate from other pos- 
sible influences on student performance and subsequent postsecondary educational 
and employment experiences. The levels of analyses proposed above help to control 
some sources of influence on the outcomes, particularly degree of program implemen- 
tation and degree of student use of the program. In addition, the more similiar stu- 
dents in a comparison group are to the participating students, the less likely it is that 
selected student-related characteristics can account for differences in outcomes (as 
opposed to the actual influence of the program itself). In examining the comparative 
outcome results, every effort should be made to consider other possible explanations 
for the differences (or lack of differences) yielded, to be certain that the program's 
impact is being well defined by the comparative outcome results. 
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The impact evaluation results can be presented and reported in different ways, 
depending on the intended audiences and purposes of the evaluation. A full report 
presenting the evaluation design, methodology, and results will be necessary but will 
be read by only a small number of interested program officials. Such a report is most 
useful as a resource or backup guide. Most audiences, such as program r- -f, board 
members, participants, and other interested parties, would benefit more from a brief, 
clearly written synopsis of the key findings that are related to the program's mission 
and design. A similar, possibly more focused brief highlighting key findings and sig- 
nificance of the program's design and impact can be shared with key funders and the 
press for public relations purposes. 

In writing impact evaluation results, it is important to consider the interest and 
methodological expertise of the intended audiences. Program managers benefit greatly 
from descriptive information on who is served and the degree to which the partici- 
pants make use of program services. Few audiences can make much use of high-level 
statistical analyses and presentations. While such analyses may be necessary to investi- 
gate and uncover a program's comparative impact, the simplest form of statistical pre- 
sentation and method of interpreting results is most useful for the broadest range of 
audiences. 

The results of an evaluation, either process or outcome, can be the basis for positive 
public relations within the program and externally for recruitment, fundraising, and 
dissemination. Good results should be shared among staff, participants, and other key 
constituencies, particularly the employers involved, to compliment their contributions 
and to reward their efforts and accomplishments. 



Evaluation Management 



Management of an evaluation entails assigning responsibility for its conduct and pro- 
viding the necessary support and resources for valid and reliable data collection and 
analysis. An evaluation, however, should not interfere with program operations or 
overburden staff and participants unnecessarily. Therefore, every effort should be 
made to devise ways of tailoring evaluation data collection to existing management 
practices, program documentation procedures, and other related activities. Finally, 
program staff and officials should consider program evaluation to be an integral part 
of program management and should look to the evaluation design and its administration 
as a way of being informed about the appropriateness and effectiveness of the program 
model, its implementation, and its service delivery strategies. 



Process and impact evaluations can be undertaken by program staff and administra- 
tion as part of their ongoing responsibilities or be contracted out to an independent 
agency or individual. Deciding who conducts the evaluation depends upon program 
staff availability and capacity as well as the need for an independent, unbiased assess- 
ment of the program. Assigning program staff to conduct the ev 'luation has several 
advantages. First, it encourages staff to consider evaluation as part of program man- 
agement, rather than an independent activity (which could be ignored). Second, it 
encourages staff to use the evaluation process and results in problem solving around 
program design, implementation, and operations. 

However, an outside evaluator should be used when existing program staff lack the 
time and expertise to undertake the evaluation or when an unbiased evaluation is 
needed to demonstrate a program's effectiveness to a wider audience. The results will 
seem to be less potentially biased if determined by an independent, unaffiliated evalu- 
ator, than if prepared by program staff. 

Besides determining who conducts the evaluation, a minimally intrusive process for 
conducting the evaluation needs to be designed. As noted above, evaluation data col- 
lection can be incorporated in several ways into existing program documentation pro- 
cedures or, at a minimum, scheduled in a way that is least intrusive on the program. 
For example, pre- and postprogram surveys on participant attitudes and expectations 
could be part of enrollment and exit forms for program participation. If this is not 
feasible, participants can be surveyed during nonclass time that does not compete with 
other testing or special events. 

The design of most school-to-work programs and the multiconstituency partnership 
that they entail add to the challenge of evaluation management. Most, like career 
academies and youth apprenticeship programs, deliver services through two or more 
institutions — the schools and the workplace. Each institution may have its own poli- 
cies and procedures governing the conduct of evaluation (such as the need for parental 
consent, confidentiality of information and records) and the method for conducting 
various data collection activities. Thus, planning and managing the evaluation require 
addressing these questions with each participating institution and providing feedback 
on the evaluation's process to ensure full participation of each institutional stakeholder. 

Finally, the evaluation should not be undertaken until all core program officials and 
staff understand the purpose of and need for the evaluation and consider ways in 
which it can be adapted to help them in their program operations and management. 
At the very least, all core program staff must support the evaluation to facilitate valid 
and reliable data collection and help collect necessary program documentation. In 
addition, program staff must be assured that the evaluation is not a punitive assessment 
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of their own roles and responsibility. Finally, the evaluation should be managed in 
such a way that it can be a creative and rewarding experience for staff and administra- 
tion as they use the evaluation design and data collection to test some of their own 
questions and assumptions about how well the program is operating. 
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